{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Final Model - Funk SVD.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "AI8iUPuAgW34",
        "VLyHpP9GcNI8",
        "ZecYlb32aPRG",
        "jXXbIrp8oniV"
      ],
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dean-sh/Movie-Ratings-Collaborating-Filltering/blob/master/Final%20Model%20-%20Funk%20SVD.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AI8iUPuAgW34",
        "colab_type": "text"
      },
      "source": [
        "MovieLens Recommendations - Final Model - Funk SVD\n",
        "=============================================\n",
        "## Dean Shabi, Dedi Kovatch, July 2019 \n",
        "## Final Project for TCDS - Technion Data Science Specialization\n",
        "\n",
        "MovieLens data sets were collected by the GroupLens Research Project\n",
        "at the University of Minnesota.\n",
        " \n",
        "This data set consists of: \n",
        "\t* 100,000 ratings (1-5) from 943 users on 1682 movies. \n",
        "\t* Each user has rated at least 20 movies. \n",
        "        * Simple demographic info for the users (age, gender, occupation, zip)\n",
        "\n",
        "The data was collected through the MovieLens web site\n",
        "(movielens.umn.edu) during the seven-month period from September 19th, \n",
        "1997 through April 22nd, 1998. This data has been cleaned up - users\n",
        "who had less than 20 ratings or did not have complete demographic\n",
        "information were removed from this data set. Detailed descriptions of\n",
        "the data file can be found at the end of this file.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GueKjIMV6KCW"
      },
      "source": [
        "# Funk SVD - Final Model\n",
        "\n",
        "Based on Repo by **Geoffrey Bolmier**, Université Paul Sabatier.\n",
        "His repo implements Simon Funk's SVD algorithm fast and effieciently.\n",
        "\n",
        "https://github.com/gbolmier/funk-svd\n",
        "\n",
        "We got amazing recommendations with this algorithm and this is the final one we picked.\n",
        "\n",
        "### **We also brought the model to 35 real users with a survey, where we got their own recommendations and got some good feedback.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4a40rehOQdb1",
        "colab_type": "text"
      },
      "source": [
        "## Download Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fZF2ZZRoUawI",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "https://github.com/gbolmier/funk-svd/blob/master/funk_svd/svd.py"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y4Dp1OATKgWZ",
        "colab_type": "code",
        "outputId": "d53b99b0-e0e6-4c9b-fb85-44ab3d0facdf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "!pip install git+https://github.com/gbolmier/funk-svd"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting git+https://github.com/gbolmier/funk-svd\n",
            "  Cloning https://github.com/gbolmier/funk-svd to /tmp/pip-req-build-n3mik0ke\n",
            "  Running command git clone -q https://github.com/gbolmier/funk-svd /tmp/pip-req-build-n3mik0ke\n",
            "Requirement already satisfied: numba>=0.38.0 in /usr/local/lib/python3.6/dist-packages (from funk-svd==0.0.1.dev1) (0.40.1)\n",
            "Requirement already satisfied: numpy>=1.14.3 in /usr/local/lib/python3.6/dist-packages (from funk-svd==0.0.1.dev1) (1.16.4)\n",
            "Requirement already satisfied: pandas>=0.23.0 in /usr/local/lib/python3.6/dist-packages (from funk-svd==0.0.1.dev1) (0.24.2)\n",
            "Requirement already satisfied: llvmlite>=0.25.0dev0 in /usr/local/lib/python3.6/dist-packages (from numba>=0.38.0->funk-svd==0.0.1.dev1) (0.29.0)\n",
            "Requirement already satisfied: python-dateutil>=2.5.0 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.23.0->funk-svd==0.0.1.dev1) (2.5.3)\n",
            "Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas>=0.23.0->funk-svd==0.0.1.dev1) (2018.9)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.5.0->pandas>=0.23.0->funk-svd==0.0.1.dev1) (1.12.0)\n",
            "Building wheels for collected packages: funk-svd\n",
            "  Building wheel for funk-svd (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-0p_6suex/wheels/66/f1/cb/e8147525b73388cc0bd5588c915e731ab65aba9a968e3ba455\n",
            "Successfully built funk-svd\n",
            "Installing collected packages: funk-svd\n",
            "Successfully installed funk-svd-0.0.1.dev1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pd5v5QEDKiX2",
        "colab_type": "code",
        "outputId": "0798b91a-139d-450e-ca3c-7b70a24fd6cb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        }
      },
      "source": [
        "from funk_svd.dataset import fetch_ml20m_ratings\n",
        "from funk_svd import SVD\n",
        "\n",
        "from sklearn.metrics import mean_absolute_error\n",
        "\n",
        "#20m dataset\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import zipfile\n",
        "import urllib.request\n",
        "\n",
        "Dataset20m = True\n",
        "Download_new_dataset = True\n",
        "\n",
        "if Download_new_dataset:\n",
        "    if Dataset20m:\n",
        "        print(\"Downloading 20-m movielens data...\")\n",
        "\n",
        "        urllib.request.urlretrieve(\"http://files.grouplens.org/datasets/movielens/ml-20m.zip\", \"movielens20m.zip\")\n",
        "\n",
        "        zip_ref = zipfile.ZipFile('movielens20m.zip', \"r\")\n",
        "        zip_ref.extractall()\n",
        "        print(\"Downloaded the 20-m movielens!\")\n",
        "\n",
        "        movies_df = pd.read_csv('ml-20m/movies.csv',names= ['i_id', 'title', 'genres'], sep=',', encoding='latin-1')\n",
        "        movies_df.drop([0], inplace=True)\n",
        "        movies_df['i_id'] = movies_df['i_id'].apply(pd.to_numeric)\n",
        "\n",
        "        # Create one merged DataFrame containing all the movielens data.\n",
        "\n",
        "        df = fetch_ml20m_ratings()\n",
        "        movielens18 = df.copy()\n",
        "\n",
        "    else:\n",
        "        print(\"Downloading 100K latest movielens data...\")\n",
        "\n",
        "        urllib.request.urlretrieve(\"http://files.grouplens.org/datasets/movielens/ml-latest-small.zip\", \"movielens.zip\")\n",
        "\n",
        "        zip_ref = zipfile.ZipFile('movielens.zip', \"r\")\n",
        "        zip_ref.extractall()\n",
        "\n",
        "        ratings_df = pd.read_csv('ml-latest-small/ratings.csv', names=['u_id', 'i_id', 'rating', 'timestamp'], sep=',', encoding='latin-1', header = None)\n",
        "        ratings_df.drop([0], inplace=True)\n",
        "        ratings_df=ratings_df.apply(pd.to_numeric)\n",
        "\n",
        "\n",
        "        movies_df = pd.read_csv('ml-latest-small/movies.csv',names= ['i_id', 'title', 'genres'], sep=',', encoding='latin-1')\n",
        "        movies_df.drop([0], inplace=True)\n",
        "        movies_df['i_id'] = movies_df['i_id'].apply(pd.to_numeric)\n",
        "\n",
        "        # Create one merged DataFrame containing all the movielens data.\n",
        "        movielens18 = ratings_df.merge(movies_df, on='i_id')\n",
        "        movielens18.drop(axis = 1, columns = ['title', 'genres', 'timestamp'], inplace = True)\n",
        "        movielens18.columns = ['u_id', 'i_id', 'rating']"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading 20-m movielens data...\n",
            "Downloaded the 20-m movielens!\n",
            "Downloading data...\n",
            "Unzipping data...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zV0mnIBb40H6",
        "colab_type": "text"
      },
      "source": [
        "## HyperParameter Search"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1XEAMn8wIZF6",
        "colab_type": "text"
      },
      "source": [
        "### Random Search"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WdsTpD0uf5pZ",
        "colab_type": "text"
      },
      "source": [
        "Sampling Learning rate, lambda, and number of latent features\\factors from a uniform distribution, about 100 times.\n",
        "In each iteration, the parameters as well as the RMSE and MAE are being saved in the results dataframe."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qohXY7cfB8t2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.metrics import mean_squared_error, mean_absolute_error\n",
        "# movielens18.drop(columns = 'timestamp', inplace = True)\n",
        "\n",
        "\n",
        "train = movielens18.sample(frac=0.8)\n",
        "val = movielens18.drop(train.index.tolist()).sample(frac=0.5, random_state=8)\n",
        "test = movielens18.drop(train.index.tolist()).drop(val.index.tolist())\n",
        "\n",
        "iterations = 100\n",
        "\n",
        "def sample_params():\n",
        "    lr = np.random.uniform(low = 0.001, high = 0.1,  size = 1)[0]\n",
        "    reg = np.random.uniform(low = 0.001, high = 0.1,  size = 1)[0]\n",
        "#     factors = np.random.randint(low = 100, high = 500,  size = 1)[0]\n",
        "    factors = 300\n",
        "    return lr, reg, factors"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LKnSkTniD1jS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a6WhwhEF4zUR",
        "colab_type": "code",
        "outputId": "18c3c211-2129-44ce-8c5c-81ee466972c6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 225
        }
      },
      "source": [
        "results = []\n",
        "for i in range(iterations):\n",
        "    \n",
        "    lr, reg, factors = sample_params()\n",
        "    \n",
        "    svd = SVD(learning_rate=lr, regularization=reg, n_epochs=5, n_factors=factors,\n",
        "              min_rating=0.5, max_rating=5)\n",
        "    \n",
        "    svd.fit(X=train, X_val=val, early_stopping=True, shuffle=False)\n",
        "\n",
        "    pred = svd.predict(test)\n",
        "    mae = mean_absolute_error(test[\"rating\"], pred)\n",
        "    rmse = np.sqrt(mean_squared_error(test[\"rating\"], pred))\n",
        "    print(\"Test MAE:  {:.2f}\".format(mae))\n",
        "    print(\"Test RMSE: {:.2f}\".format(rmse))\n",
        "    print('{} factors, {} lr, {} reg'.format(factors, lr, reg))\n",
        "    results.append([rmse, mae, lr, reg, factors])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Preprocessing data...\n",
            "\n",
            "Epoch 1/5  | val_loss: 0.74 - val_rmse: 0.86 - val_mae: 0.66 - took 18.8 sec\n",
            "Epoch 2/5  | val_loss: 0.71 - val_rmse: 0.85 - val_mae: 0.65 - took 17.2 sec\n",
            "Epoch 3/5  | val_loss: 0.70 - val_rmse: 0.84 - val_mae: 0.64 - took 17.5 sec\n",
            "Epoch 4/5  | val_loss: 0.69 - val_rmse: 0.83 - val_mae: 0.64 - took 17.7 sec\n",
            "Epoch 5/5  | val_loss: 0.68 - val_rmse: 0.82 - val_mae: 0.63 - took 17.6 sec\n",
            "\n",
            "Training took 1 min and 41 sec\n",
            "Test MAE:  0.63\n",
            "Test RMSE: 0.82\n",
            "300 factors, 0.03726422643591702 lr, 0.07328935967245179 reg\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g_YjnOmvFNw0",
        "colab_type": "code",
        "outputId": "2c0d6bef-f9d1-4281-b6e3-c2f4bc35cc61",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 97
        }
      },
      "source": [
        "results = pd.DataFrame(results)\n",
        "results.columns = ['RMSE', 'MAE', 'learningRate', 'Lambda', 'NumOfFeatures']\n",
        "\n",
        "print(\"Random search results:\")\n",
        "results.sort_values(by='RMSE').head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Random search results:\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>RMSE</th>\n",
              "      <th>MAE</th>\n",
              "      <th>learningRate</th>\n",
              "      <th>Lambda</th>\n",
              "      <th>NumOfFeatures</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.82485</td>\n",
              "      <td>0.632915</td>\n",
              "      <td>0.037264</td>\n",
              "      <td>0.073289</td>\n",
              "      <td>300</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      RMSE       MAE  learningRate    Lambda  NumOfFeatures\n",
              "0  0.82485  0.632915      0.037264  0.073289            300"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I4uJivTsHOF-",
        "colab_type": "text"
      },
      "source": [
        "### Training the best model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wbKSRygvHNxR",
        "colab_type": "code",
        "outputId": "effd09c0-132e-43db-c56d-b528b9e7f53f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 487
        }
      },
      "source": [
        "lr, reg, factors = (0.007, 0.03, 90)\n",
        "\n",
        "svd = SVD(learning_rate=lr, regularization=reg, n_epochs=200, n_factors=factors,\n",
        "          min_rating=0.5, max_rating=5)\n",
        "\n",
        "svd.fit(X=train, X_val=val, early_stopping=True, shuffle=False)\n",
        "\n",
        "pred = svd.predict(test)\n",
        "mae = mean_absolute_error(test[\"rating\"], pred)\n",
        "rmse = np.sqrt(mean_squared_error(test[\"rating\"], pred))\n",
        "print(\"Test MAE:  {:.2f}\".format(mae))\n",
        "print(\"Test RMSE: {:.2f}\".format(rmse))\n",
        "print('{} factors, {} lr, {} reg'.format(factors, lr, reg))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Preprocessing data...\n",
            "\n",
            "Epoch 1/200  | val_loss: 0.77 - val_rmse: 0.87 - val_mae: 0.67 - took 6.9 sec\n",
            "Epoch 2/200  | val_loss: 0.75 - val_rmse: 0.86 - val_mae: 0.67 - took 6.2 sec\n",
            "Epoch 3/200  | val_loss: 0.73 - val_rmse: 0.85 - val_mae: 0.66 - took 6.2 sec\n",
            "Epoch 4/200  | val_loss: 0.71 - val_rmse: 0.84 - val_mae: 0.65 - took 6.1 sec\n",
            "Epoch 5/200  | val_loss: 0.70 - val_rmse: 0.83 - val_mae: 0.64 - took 6.1 sec\n",
            "Epoch 6/200  | val_loss: 0.68 - val_rmse: 0.83 - val_mae: 0.63 - took 6.1 sec\n",
            "Epoch 7/200  | val_loss: 0.67 - val_rmse: 0.82 - val_mae: 0.63 - took 6.1 sec\n",
            "Epoch 8/200  | val_loss: 0.66 - val_rmse: 0.81 - val_mae: 0.62 - took 6.1 sec\n",
            "Epoch 9/200  | val_loss: 0.65 - val_rmse: 0.81 - val_mae: 0.62 - took 6.1 sec\n",
            "Epoch 10/200 | val_loss: 0.64 - val_rmse: 0.80 - val_mae: 0.61 - took 6.1 sec\n",
            "Epoch 11/200 | val_loss: 0.64 - val_rmse: 0.80 - val_mae: 0.61 - took 6.1 sec\n",
            "Epoch 12/200 | val_loss: 0.63 - val_rmse: 0.80 - val_mae: 0.61 - took 6.1 sec\n",
            "Epoch 13/200 | val_loss: 0.63 - val_rmse: 0.79 - val_mae: 0.61 - took 6.1 sec\n",
            "Epoch 14/200 | val_loss: 0.62 - val_rmse: 0.79 - val_mae: 0.60 - took 6.1 sec\n",
            "Epoch 15/200 | val_loss: 0.62 - val_rmse: 0.79 - val_mae: 0.60 - took 6.1 sec\n",
            "Epoch 16/200 | val_loss: 0.62 - val_rmse: 0.79 - val_mae: 0.60 - took 6.0 sec\n",
            "Epoch 17/200 | val_loss: 0.61 - val_rmse: 0.78 - val_mae: 0.60 - took 6.0 sec\n",
            "Epoch 18/200 | val_loss: 0.61 - val_rmse: 0.78 - val_mae: 0.60 - took 6.0 sec\n",
            "Epoch 19/200 | val_loss: 0.61 - val_rmse: 0.78 - val_mae: 0.60 - took 6.1 sec\n",
            "Epoch 20/200 | val_loss: 0.61 - val_rmse: 0.78 - val_mae: 0.60 - took 6.0 sec\n",
            "Epoch 21/200 | val_loss: 0.61 - val_rmse: 0.78 - val_mae: 0.59 - took 6.1 sec\n",
            "\n",
            "Training took 2 min and 18 sec\n",
            "Test MAE:  0.59\n",
            "Test RMSE: 0.78\n",
            "90 factors, 0.007 lr, 0.03 reg\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V9ZRdSVBITP1",
        "colab_type": "text"
      },
      "source": [
        "## User Recommendations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mzWcIK2Donh-"
      },
      "source": [
        "### Random Search"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "328ac399-ab9b-41c2-f894-6cb5a0ee9006",
        "id": "7uSO1v7aOjww",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 185
        }
      },
      "source": [
        "#Adding our own ratings\n",
        "\n",
        "n_m = len(movielens18.i_id.unique())\n",
        "\n",
        "#  Initialize my ratings\n",
        "my_ratings = np.zeros(n_m)\n",
        "\n",
        "\n",
        "my_ratings[4993] = 5\n",
        "my_ratings[1080] = 5\n",
        "my_ratings[260] = 5\n",
        "my_ratings[4896] = 5\n",
        "my_ratings[1196] = 5\n",
        "my_ratings[1210] = 5\n",
        "my_ratings[2628] = 5\n",
        "my_ratings[5378] = 5\n",
        "\n",
        "print('User ratings:')\n",
        "print('-----------------')\n",
        "\n",
        "for i, val in enumerate(my_ratings):\n",
        "    if val > 0:\n",
        "        print('Rated %d stars: %s' % (val, movies_df.loc[movies_df.i_id==i].title.values))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "User ratings:\n",
            "-----------------\n",
            "Rated 5 stars: ['Star Wars: Episode IV - A New Hope (1977)']\n",
            "Rated 5 stars: [\"Monty Python's Life of Brian (1979)\"]\n",
            "Rated 5 stars: ['Star Wars: Episode V - The Empire Strikes Back (1980)']\n",
            "Rated 5 stars: ['Star Wars: Episode VI - Return of the Jedi (1983)']\n",
            "Rated 5 stars: ['Star Wars: Episode I - The Phantom Menace (1999)']\n",
            "Rated 5 stars: [\"Harry Potter and the Sorcerer's Stone (a.k.a. Harry Potter and the Philosopher's Stone) (2001)\"]\n",
            "Rated 5 stars: ['Lord of the Rings: The Fellowship of the Ring, The (2001)']\n",
            "Rated 5 stars: ['Star Wars: Episode II - Attack of the Clones (2002)']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PLGed5pcUMho",
        "colab_type": "code",
        "outputId": "a0eed382-ef83-4cc9-fa7b-0a433e6ab686",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(\"Adding your recommendations!\")\n",
        "items_id = [item[0] for item in np.argwhere(my_ratings>0)]\n",
        "ratings_list = my_ratings[np.where(my_ratings>0)]\n",
        "user_id = np.asarray([0] * len(ratings_list))\n",
        "\n",
        "user_ratings = pd.DataFrame(list(zip(user_id, items_id, ratings_list)), columns=['u_id', 'i_id', 'rating'])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Adding your recommendations!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AjdqdLe3RM7X",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "try:\n",
        "    movielens18 = movielens18.drop(columns=['timestamp'])\n",
        "except:\n",
        "    pass\n",
        "data_with_user = movielens18.append(user_ratings, ignore_index=True)\n",
        "\n",
        "train_user = data_with_user.sample(frac=0.8)\n",
        "val_user = data_with_user.drop(train_user.index.tolist()).sample(frac=0.5, random_state=8)\n",
        "test_user = data_with_user.drop(train_user.index.tolist()).drop(val_user.index.tolist())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nyOylYX3Qlis",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from itertools import product\n",
        "def funk_svd_predict(userID, data_with_user, movies_df):\n",
        "    userID = [userID]\n",
        "\n",
        "    # all_users = data_with_user.u_id.unique()\n",
        "    all_movies = data_with_user.i_id.unique()\n",
        "    recommendations = pd.DataFrame(list(product(userID, all_movies)), columns=['u_id', 'i_id'])\n",
        "\n",
        "    #Getting predictions for the selected userID\n",
        "    pred_train = svd.predict(recommendations)\n",
        "    recommendations['prediction'] = pred_train\n",
        "    recommendations.head(10)\n",
        "\n",
        "    sorted_user_predictions = recommendations.sort_values(by='prediction', ascending=False)\n",
        "\n",
        "    user_ratings = data_with_user[data_with_user.u_id == userID[0]]\n",
        "    user_ratings.columns = ['u_id',\t'i_id', 'rating']\n",
        "    # Recommend the highest predicted rating movies that the user hasn't seen yet.\n",
        "    recommendations = movies_df[~movies_df['i_id'].isin(user_ratings['i_id'])].\\\n",
        "        merge(pd.DataFrame(sorted_user_predictions).reset_index(drop=True), how = 'inner', left_on = 'i_id', right_on = 'i_id').\\\n",
        "        sort_values(by='prediction', ascending = False).drop(['i_id'],axis=1)\n",
        "\n",
        "    rated_df = movies_df[movies_df['i_id'].isin(user_ratings['i_id'])].\\\n",
        "        merge(pd.DataFrame(data_with_user).reset_index(drop=True), how = 'inner', left_on = 'i_id', right_on = 'i_id')\n",
        "    rated_df = rated_df.loc[rated_df.u_id==userID[0]].sort_values(by='rating', ascending = False)\n",
        "    \n",
        "    return recommendations, rated_df"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ur24MXt8oniG",
        "colab": {}
      },
      "source": [
        "from sklearn.metrics import mean_squared_error, mean_absolute_error\n",
        "\n",
        "iterations = 1\n",
        "\n",
        "def sample_params():\n",
        "    lr = np.random.uniform(low = 0.001, high = 0.01,  size = 1)[0]\n",
        "    reg = np.random.uniform(low = 0.001, high = 0.1,  size = 1)[0]\n",
        "    factors = np.random.randint(low = 30, high = 200,  size = 1)[0]\n",
        "#     factors = 300\n",
        "    return lr, reg, factors"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b2KBrAdBpzf6",
        "colab_type": "code",
        "outputId": "623185ed-a117-453f-8aec-18a0702143ee",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "#Checking Num Movies to see if this is the 20M data (should be 26K movies)\n",
        "len(movielens18.i_id.unique())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "26744"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "bC59-QOwoniL",
        "colab": {}
      },
      "source": [
        "results = []\n",
        "for i in range(iterations):\n",
        "    \n",
        "    lr, reg, factors = sample_params()\n",
        "    \n",
        "    svd = SVD(learning_rate=lr, regularization=reg, n_epochs=20, n_factors=factors,\n",
        "              min_rating=0.5, max_rating=5)\n",
        "    \n",
        "    print('{} factors, {} lr, {} reg'.format(factors, lr, reg))\n",
        "\n",
        "    svd.fit(X=train_user, X_val=val_user, early_stopping=True, shuffle=False)\n",
        "\n",
        "    pred = svd.predict(test_user)\n",
        "    mae = mean_absolute_error(test_user[\"rating\"], pred)\n",
        "    rmse = np.sqrt(mean_squared_error(test_user[\"rating\"], pred))\n",
        "    print(\"Test MAE:  {:.2f}\".format(mae))\n",
        "    print(\"Test RMSE: {:.2f}\\n\".format(rmse))\n",
        "    results.append([rmse, mae, lr, reg, factors])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "90935ad7-cb72-488e-832e-ee318993615a",
        "id": "ScdsbaO7oniQ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 95
        }
      },
      "source": [
        "results = pd.DataFrame(results)\n",
        "results.columns = ['RMSE', 'MAE', 'learningRate', 'Lambda', 'NumOfFeatures']\n",
        "\n",
        "print(\"Random search results:\")\n",
        "results.sort_values(by='RMSE').head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Random search results:\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>RMSE</th>\n",
              "      <th>MAE</th>\n",
              "      <th>learningRate</th>\n",
              "      <th>Lambda</th>\n",
              "      <th>NumOfFeatures</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.833328</td>\n",
              "      <td>0.640036</td>\n",
              "      <td>0.00615</td>\n",
              "      <td>0.090192</td>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       RMSE       MAE  learningRate    Lambda  NumOfFeatures\n",
              "0  0.833328  0.640036       0.00615  0.090192             30"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "jXXbIrp8oniV"
      },
      "source": [
        "### Training the best model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Y-g3AXYnoniW",
        "outputId": "0e70c06e-f551-4f4e-abbb-67a4031a60fa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 487
        }
      },
      "source": [
        "lr, reg, factors = (0.007, 0.03, 90)\n",
        "epochs = 50\n",
        "\n",
        "svd = SVD(learning_rate=lr, regularization=reg, n_epochs=epochs, n_factors=factors,\n",
        "          min_rating=0.5, max_rating=5)\n",
        "\n",
        "svd.fit(X=train_user, X_val=val_user, early_stopping=True, shuffle=False)\n",
        "\n",
        "pred = svd.predict(test_user)\n",
        "mae = mean_absolute_error(test_user[\"rating\"], pred)\n",
        "rmse = np.sqrt(mean_squared_error(test_user[\"rating\"], pred))\n",
        "print(\"Test MAE:  {:.2f}\".format(mae))\n",
        "print(\"Test RMSE: {:.2f}\".format(rmse))\n",
        "print('{} factors, {} lr, {} reg'.format(factors, lr, reg))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Preprocessing data...\n",
            "\n",
            "Epoch 1/50  | val_loss: 0.76 - val_rmse: 0.87 - val_mae: 0.67 - took 6.1 sec\n",
            "Epoch 2/50  | val_loss: 0.75 - val_rmse: 0.86 - val_mae: 0.66 - took 6.2 sec\n",
            "Epoch 3/50  | val_loss: 0.73 - val_rmse: 0.85 - val_mae: 0.66 - took 6.2 sec\n",
            "Epoch 4/50  | val_loss: 0.71 - val_rmse: 0.84 - val_mae: 0.65 - took 6.1 sec\n",
            "Epoch 5/50  | val_loss: 0.69 - val_rmse: 0.83 - val_mae: 0.64 - took 6.1 sec\n",
            "Epoch 6/50  | val_loss: 0.68 - val_rmse: 0.83 - val_mae: 0.63 - took 6.1 sec\n",
            "Epoch 7/50  | val_loss: 0.67 - val_rmse: 0.82 - val_mae: 0.63 - took 6.1 sec\n",
            "Epoch 8/50  | val_loss: 0.66 - val_rmse: 0.81 - val_mae: 0.62 - took 6.1 sec\n",
            "Epoch 9/50  | val_loss: 0.65 - val_rmse: 0.81 - val_mae: 0.62 - took 6.1 sec\n",
            "Epoch 10/50 | val_loss: 0.64 - val_rmse: 0.80 - val_mae: 0.61 - took 6.3 sec\n",
            "Epoch 11/50 | val_loss: 0.64 - val_rmse: 0.80 - val_mae: 0.61 - took 6.3 sec\n",
            "Epoch 12/50 | val_loss: 0.63 - val_rmse: 0.79 - val_mae: 0.61 - took 6.1 sec\n",
            "Epoch 13/50 | val_loss: 0.63 - val_rmse: 0.79 - val_mae: 0.61 - took 6.3 sec\n",
            "Epoch 14/50 | val_loss: 0.62 - val_rmse: 0.79 - val_mae: 0.60 - took 6.3 sec\n",
            "Epoch 15/50 | val_loss: 0.62 - val_rmse: 0.79 - val_mae: 0.60 - took 6.3 sec\n",
            "Epoch 16/50 | val_loss: 0.62 - val_rmse: 0.79 - val_mae: 0.60 - took 6.1 sec\n",
            "Epoch 17/50 | val_loss: 0.61 - val_rmse: 0.78 - val_mae: 0.60 - took 6.1 sec\n",
            "Epoch 18/50 | val_loss: 0.61 - val_rmse: 0.78 - val_mae: 0.60 - took 6.1 sec\n",
            "Epoch 19/50 | val_loss: 0.61 - val_rmse: 0.78 - val_mae: 0.60 - took 6.1 sec\n",
            "Epoch 20/50 | val_loss: 0.61 - val_rmse: 0.78 - val_mae: 0.59 - took 6.1 sec\n",
            "Epoch 21/50 | val_loss: 0.61 - val_rmse: 0.78 - val_mae: 0.59 - took 6.1 sec\n",
            "\n",
            "Training took 2 min and 14 sec\n",
            "Test MAE:  0.59\n",
            "Test RMSE: 0.78\n",
            "90 factors, 0.007 lr, 0.03 reg\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FjgiDqtepYhz",
        "colab_type": "text"
      },
      "source": [
        "### Predictions:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0u44LuNrMFrZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from itertools import product\n",
        "def funk_svd_predict(userID, data_with_user, movies_df):\n",
        "    userID = [userID]\n",
        "\n",
        "    # all_users = data_with_user.u_id.unique()\n",
        "    all_movies = data_with_user.i_id.unique()\n",
        "    recommendations = pd.DataFrame(list(product(userID, all_movies)), columns=['u_id', 'i_id'])\n",
        "\n",
        "    #Getting predictions for the selected userID\n",
        "    pred_train = svd.predict(recommendations)\n",
        "    recommendations['prediction'] = pred_train\n",
        "    recommendations.head(10)\n",
        "\n",
        "    sorted_user_predictions = recommendations.sort_values(by='prediction', ascending=False)\n",
        "\n",
        "    user_ratings = data_with_user[data_with_user.u_id == userID[0]]\n",
        "    user_ratings.columns = ['u_id',\t'i_id', 'rating']\n",
        "    # Recommend the highest predicted rating movies that the user hasn't seen yet.\n",
        "    recommendations = movies_df[~movies_df['i_id'].isin(user_ratings['i_id'])].\\\n",
        "        merge(pd.DataFrame(sorted_user_predictions).reset_index(drop=True), how = 'inner', left_on = 'i_id', right_on = 'i_id').\\\n",
        "        sort_values(by='prediction', ascending = False)#.drop(['i_id'],axis=1)\n",
        "\n",
        "    rated_df = movies_df[movies_df['i_id'].isin(user_ratings['i_id'])].\\\n",
        "        merge(pd.DataFrame(data_with_user).reset_index(drop=True), how = 'inner', left_on = 'i_id', right_on = 'i_id')\n",
        "    rated_df = rated_df.loc[rated_df.u_id==userID[0]].sort_values(by='rating', ascending = False)\n",
        "    \n",
        "    return recommendations, rated_df"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "pyyqY5aGOjw3",
        "colab": {}
      },
      "source": [
        "recommendations, rated_df = funk_svd_predict(0, data_with_user, movies_df)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Um20ZaK5gnWH",
        "colab_type": "code",
        "outputId": "7d94d146-1f4b-4130-fde2-7e2682a5eff4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 284
        }
      },
      "source": [
        "rated_df"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>i_id</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "      <th>u_id</th>\n",
              "      <th>rating</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>54502</th>\n",
              "      <td>260</td>\n",
              "      <td>Star Wars: Episode IV - A New Hope (1977)</td>\n",
              "      <td>Action|Adventure|Sci-Fi</td>\n",
              "      <td>0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75060</th>\n",
              "      <td>1080</td>\n",
              "      <td>Monty Python's Life of Brian (1979)</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>120374</th>\n",
              "      <td>1196</td>\n",
              "      <td>Star Wars: Episode V - The Empire Strikes Back...</td>\n",
              "      <td>Action|Adventure|Sci-Fi</td>\n",
              "      <td>0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>167214</th>\n",
              "      <td>1210</td>\n",
              "      <td>Star Wars: Episode VI - Return of the Jedi (1983)</td>\n",
              "      <td>Action|Adventure|Sci-Fi</td>\n",
              "      <td>0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>196789</th>\n",
              "      <td>2628</td>\n",
              "      <td>Star Wars: Episode I - The Phantom Menace (1999)</td>\n",
              "      <td>Action|Adventure|Sci-Fi</td>\n",
              "      <td>0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>214029</th>\n",
              "      <td>4896</td>\n",
              "      <td>Harry Potter and the Sorcerer's Stone (a.k.a. ...</td>\n",
              "      <td>Adventure|Children|Fantasy</td>\n",
              "      <td>0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>251583</th>\n",
              "      <td>4993</td>\n",
              "      <td>Lord of the Rings: The Fellowship of the Ring,...</td>\n",
              "      <td>Adventure|Fantasy</td>\n",
              "      <td>0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>268009</th>\n",
              "      <td>5378</td>\n",
              "      <td>Star Wars: Episode II - Attack of the Clones (...</td>\n",
              "      <td>Action|Adventure|Sci-Fi|IMAX</td>\n",
              "      <td>0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        i_id                                              title  ... u_id  rating\n",
              "54502    260          Star Wars: Episode IV - A New Hope (1977)  ...    0     5.0\n",
              "75060   1080                Monty Python's Life of Brian (1979)  ...    0     5.0\n",
              "120374  1196  Star Wars: Episode V - The Empire Strikes Back...  ...    0     5.0\n",
              "167214  1210  Star Wars: Episode VI - Return of the Jedi (1983)  ...    0     5.0\n",
              "196789  2628   Star Wars: Episode I - The Phantom Menace (1999)  ...    0     5.0\n",
              "214029  4896  Harry Potter and the Sorcerer's Stone (a.k.a. ...  ...    0     5.0\n",
              "251583  4993  Lord of the Rings: The Fellowship of the Ring,...  ...    0     5.0\n",
              "268009  5378  Star Wars: Episode II - Attack of the Clones (...  ...    0     5.0\n",
              "\n",
              "[8 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 85
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gln2z24ngpWg",
        "colab_type": "code",
        "outputId": "ba8a5144-12f3-49e7-9219-d4f21594a7b1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 935
        }
      },
      "source": [
        "recommendations.head(30)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>i_id</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "      <th>u_id</th>\n",
              "      <th>prediction</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>5845</th>\n",
              "      <td>5952</td>\n",
              "      <td>Lord of the Rings: The Two Towers, The (2002)</td>\n",
              "      <td>Adventure|Fantasy</td>\n",
              "      <td>0</td>\n",
              "      <td>5.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7348</th>\n",
              "      <td>7502</td>\n",
              "      <td>Band of Brothers (2001)</td>\n",
              "      <td>Action|Drama|War</td>\n",
              "      <td>0</td>\n",
              "      <td>5.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7033</th>\n",
              "      <td>7153</td>\n",
              "      <td>Lord of the Rings: The Return of the King, The...</td>\n",
              "      <td>Action|Adventure|Drama|Fantasy</td>\n",
              "      <td>0</td>\n",
              "      <td>5.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9784</th>\n",
              "      <td>31948</td>\n",
              "      <td>Phone Box, The (Cabina, La) (1972)</td>\n",
              "      <td>Comedy|Drama|Mystery|Thriller</td>\n",
              "      <td>0</td>\n",
              "      <td>5.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2482</th>\n",
              "      <td>2571</td>\n",
              "      <td>Matrix, The (1999)</td>\n",
              "      <td>Action|Sci-Fi|Thriller</td>\n",
              "      <td>0</td>\n",
              "      <td>5.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>314</th>\n",
              "      <td>318</td>\n",
              "      <td>Shawshank Redemption, The (1994)</td>\n",
              "      <td>Crime|Drama</td>\n",
              "      <td>0</td>\n",
              "      <td>4.981206</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8965</th>\n",
              "      <td>26674</td>\n",
              "      <td>Prime Suspect (1991)</td>\n",
              "      <td>Crime|Drama|Mystery|Thriller</td>\n",
              "      <td>0</td>\n",
              "      <td>4.968790</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20463</th>\n",
              "      <td>100553</td>\n",
              "      <td>Frozen Planet (2011)</td>\n",
              "      <td>Documentary</td>\n",
              "      <td>0</td>\n",
              "      <td>4.967922</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>50</td>\n",
              "      <td>Usual Suspects, The (1995)</td>\n",
              "      <td>Crime|Mystery|Thriller</td>\n",
              "      <td>0</td>\n",
              "      <td>4.956233</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1111</th>\n",
              "      <td>1136</td>\n",
              "      <td>Monty Python and the Holy Grail (1975)</td>\n",
              "      <td>Adventure|Comedy|Fantasy</td>\n",
              "      <td>0</td>\n",
              "      <td>4.938441</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1169</th>\n",
              "      <td>1197</td>\n",
              "      <td>Princess Bride, The (1987)</td>\n",
              "      <td>Action|Adventure|Comedy|Fantasy|Romance</td>\n",
              "      <td>0</td>\n",
              "      <td>4.929875</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1259</th>\n",
              "      <td>1291</td>\n",
              "      <td>Indiana Jones and the Last Crusade (1989)</td>\n",
              "      <td>Action|Adventure</td>\n",
              "      <td>0</td>\n",
              "      <td>4.917170</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19271</th>\n",
              "      <td>96062</td>\n",
              "      <td>Jekyll (2007)</td>\n",
              "      <td>Crime|Horror|Thriller</td>\n",
              "      <td>0</td>\n",
              "      <td>4.910934</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>108</th>\n",
              "      <td>110</td>\n",
              "      <td>Braveheart (1995)</td>\n",
              "      <td>Action|Drama|War</td>\n",
              "      <td>0</td>\n",
              "      <td>4.909150</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15180</th>\n",
              "      <td>77658</td>\n",
              "      <td>Cosmos (1980)</td>\n",
              "      <td>Documentary</td>\n",
              "      <td>0</td>\n",
              "      <td>4.905674</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22048</th>\n",
              "      <td>106642</td>\n",
              "      <td>Day of the Doctor, The (2013)</td>\n",
              "      <td>Adventure|Drama|Sci-Fi</td>\n",
              "      <td>0</td>\n",
              "      <td>4.896506</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23249</th>\n",
              "      <td>111436</td>\n",
              "      <td>Jim Gaffigan: Obsessed (2014)</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>0</td>\n",
              "      <td>4.893865</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23566</th>\n",
              "      <td>112852</td>\n",
              "      <td>Guardians of the Galaxy (2014)</td>\n",
              "      <td>Action|Adventure|Sci-Fi</td>\n",
              "      <td>0</td>\n",
              "      <td>4.890249</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17563</th>\n",
              "      <td>88570</td>\n",
              "      <td>Welfare (1975)</td>\n",
              "      <td>Documentary</td>\n",
              "      <td>0</td>\n",
              "      <td>4.884352</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1170</th>\n",
              "      <td>1198</td>\n",
              "      <td>Raiders of the Lost Ark (Indiana Jones and the...</td>\n",
              "      <td>Action|Adventure</td>\n",
              "      <td>0</td>\n",
              "      <td>4.874977</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19405</th>\n",
              "      <td>96608</td>\n",
              "      <td>Runaway Brain (1995)</td>\n",
              "      <td>Animation|Comedy|Sci-Fi</td>\n",
              "      <td>0</td>\n",
              "      <td>4.863382</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21963</th>\n",
              "      <td>106334</td>\n",
              "      <td>Hollow Crown, The (2012)</td>\n",
              "      <td>Drama</td>\n",
              "      <td>0</td>\n",
              "      <td>4.853162</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3897</th>\n",
              "      <td>3996</td>\n",
              "      <td>Crouching Tiger, Hidden Dragon (Wo hu cang lon...</td>\n",
              "      <td>Action|Drama|Romance</td>\n",
              "      <td>0</td>\n",
              "      <td>4.849894</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15796</th>\n",
              "      <td>80337</td>\n",
              "      <td>Still Bill (2009)</td>\n",
              "      <td>Documentary</td>\n",
              "      <td>0</td>\n",
              "      <td>4.844832</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2235</th>\n",
              "      <td>2324</td>\n",
              "      <td>Life Is Beautiful (La Vita Ã¨ bella) (1997)</td>\n",
              "      <td>Comedy|Drama|Romance|War</td>\n",
              "      <td>0</td>\n",
              "      <td>4.843089</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12507</th>\n",
              "      <td>58559</td>\n",
              "      <td>Dark Knight, The (2008)</td>\n",
              "      <td>Action|Crime|Drama|IMAX</td>\n",
              "      <td>0</td>\n",
              "      <td>4.838164</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15275</th>\n",
              "      <td>78064</td>\n",
              "      <td>Ween Live in Chicago (2004)</td>\n",
              "      <td>Documentary</td>\n",
              "      <td>0</td>\n",
              "      <td>4.829910</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14165</th>\n",
              "      <td>71178</td>\n",
              "      <td>Street of Shame (Akasen chitai) (1956)</td>\n",
              "      <td>Drama</td>\n",
              "      <td>0</td>\n",
              "      <td>4.815295</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9151</th>\n",
              "      <td>27002</td>\n",
              "      <td>From the Earth to the Moon (1998)</td>\n",
              "      <td>Action|Documentary|Drama|Thriller</td>\n",
              "      <td>0</td>\n",
              "      <td>4.804214</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25162</th>\n",
              "      <td>120815</td>\n",
              "      <td>Patton Oswalt: Werewolves and Lollipops (2007)</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>0</td>\n",
              "      <td>4.800932</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         i_id  ... prediction\n",
              "5845     5952  ...   5.000000\n",
              "7348     7502  ...   5.000000\n",
              "7033     7153  ...   5.000000\n",
              "9784    31948  ...   5.000000\n",
              "2482     2571  ...   5.000000\n",
              "314       318  ...   4.981206\n",
              "8965    26674  ...   4.968790\n",
              "20463  100553  ...   4.967922\n",
              "49         50  ...   4.956233\n",
              "1111     1136  ...   4.938441\n",
              "1169     1197  ...   4.929875\n",
              "1259     1291  ...   4.917170\n",
              "19271   96062  ...   4.910934\n",
              "108       110  ...   4.909150\n",
              "15180   77658  ...   4.905674\n",
              "22048  106642  ...   4.896506\n",
              "23249  111436  ...   4.893865\n",
              "23566  112852  ...   4.890249\n",
              "17563   88570  ...   4.884352\n",
              "1170     1198  ...   4.874977\n",
              "19405   96608  ...   4.863382\n",
              "21963  106334  ...   4.853162\n",
              "3897     3996  ...   4.849894\n",
              "15796   80337  ...   4.844832\n",
              "2235     2324  ...   4.843089\n",
              "12507   58559  ...   4.838164\n",
              "15275   78064  ...   4.829910\n",
              "14165   71178  ...   4.815295\n",
              "9151    27002  ...   4.804214\n",
              "25162  120815  ...   4.800932\n",
              "\n",
              "[30 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 89
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RPN0p-gZBmHv",
        "colab_type": "text"
      },
      "source": [
        "## Survey real users to showcase our model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Icrsxpl0gORr",
        "colab_type": "text"
      },
      "source": [
        "**We created an online survey to aquire real user recommendations from Technion Data Science class and our friends.**\n",
        "\n",
        "The survey is in Segmanta platform and can be accessed here:\n",
        "https://surveys.segmanta.com/85fdg0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-1gAs2T-4hRV",
        "colab_type": "text"
      },
      "source": [
        "### Making A Survey"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CML4XTlogiLY",
        "colab_type": "text"
      },
      "source": [
        "**We picked the top 20 movies from the 20-m Movielens data, that have the highest rating variation and also are popular.**\n",
        "\n",
        "This was important, so that we get ratings that have high std in the data.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hBbhvnNkphlH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#setting up the list of movies for survey\n",
        "number_of_ratings = pd.DataFrame(movielens18.groupby('i_id').count().loc[:,'rating'])\n",
        "std_of_ratings = pd.DataFrame(movielens18.groupby('i_id').std().loc[:,'rating']) \n",
        "mean_of_ratings = pd.DataFrame(movielens18.groupby('i_id').mean().loc[:,'rating'])\n",
        "\n",
        "x = number_of_ratings.merge(mean_of_ratings, how='inner', on = 'i_id').merge(std_of_ratings, how='inner', on = 'i_id')\n",
        "x = x.merge(movies_df, how='inner', on = 'i_id')\n",
        "x.drop('genres', axis = 1, inplace = True)\n",
        "x.columns = ['MovieID', 'Number of Ratings', 'Avg Ratings', 'STD of ratings', 'Title']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hr457S98XCdb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "movies_pick_std = x.sort_values(by = 'Number of Ratings', ascending = False).head(150).sort_values(by = 'STD of ratings', ascending = False).head(20)\n",
        "movies_pick_mean = x.sort_values(by = 'Number of Ratings', ascending = False).head(150).sort_values(by = 'Avg Ratings', ascending = False).head(20)\n",
        "\n",
        "frames = [movies_pick_std, movies_pick_mean]\n",
        "picked_movies = pd.concat(frames)\n",
        "\n",
        "picked_movies = picked_movies.sample(frac=1, random_state=0)\n",
        "\n",
        "picked_movies = picked_movies.head(20)\n",
        "picked_movies = picked_movies.reset_index(drop=True)\n",
        "picked_movies.drop(['Number of Ratings', 'Avg Ratings', 'STD of ratings'], axis=1, inplace=True)\n",
        "\n",
        "#picked_movies"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X7M7f8H2DAxy",
        "colab_type": "text"
      },
      "source": [
        "**Adding new movies**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CLINld4UhDaB",
        "colab_type": "text"
      },
      "source": [
        "We decided to pick 24 more newer movies based on real-users survey.\n",
        "\n",
        "the \"new_survey_movies\" csv contains the additional data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P3Aqr9AcDFQR",
        "colab_type": "code",
        "outputId": "e5f9811c-6f28-4e50-b6ef-de660af98fd9",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 75
        }
      },
      "source": [
        "#Adding new movies - upload the \"new_survey_movies.csv\" here\n",
        "\n",
        "from google.colab import files\n",
        "uploaded = files.upload()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-2bd7c291-266a-43d9-abd2-a9800359b1bb\" name=\"files[]\" multiple disabled />\n",
              "     <output id=\"result-2bd7c291-266a-43d9-abd2-a9800359b1bb\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Saving new_survey_movies.csv to new_survey_movies.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y5UwqOWyEGeo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "new_movies = pd.read_csv('new_survey_movies.csv', header=None)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O72ZJdWFEPMm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for i, item in new_movies.iterrows():\n",
        "    a = movies_df[movies_df.title==item[0]]\n",
        "    line = pd.DataFrame({'MovieID':a.i_id.values[0], 'Title':a.title.values[0]}, index=[10])\n",
        "    picked_movies = picked_movies.append(line, ignore_index=False)\n",
        "\n",
        "picked_movies = picked_movies.reset_index(drop=True)\n",
        "picked_movies.MovieID = picked_movies.MovieID.astype(int)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xyC_kfZ2JqIg",
        "colab_type": "code",
        "outputId": "38c61bb9-1438-4096-e346-1745ed1d0849",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "picked_movies"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>MovieID</th>\n",
              "      <th>Title</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>50</td>\n",
              "      <td>Usual Suspects, The (1995)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>318</td>\n",
              "      <td>Shawshank Redemption, The (1994)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>912</td>\n",
              "      <td>Casablanca (1942)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2628</td>\n",
              "      <td>Star Wars: Episode I - The Phantom Menace (1999)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1517</td>\n",
              "      <td>Austin Powers: International Man of Mystery (1...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>208</td>\n",
              "      <td>Waterworld (1995)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>2959</td>\n",
              "      <td>Fight Club (1999)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>780</td>\n",
              "      <td>Independence Day (a.k.a. ID4) (1996)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>736</td>\n",
              "      <td>Twister (1996)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1198</td>\n",
              "      <td>Raiders of the Lost Ark (Indiana Jones and the...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>750</td>\n",
              "      <td>Dr. Strangelove or: How I Learned to Stop Worr...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>4226</td>\n",
              "      <td>Memento (2000)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>1197</td>\n",
              "      <td>Princess Bride, The (1987)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>1721</td>\n",
              "      <td>Titanic (1997)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>1136</td>\n",
              "      <td>Monty Python and the Holy Grail (1975)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>4973</td>\n",
              "      <td>Amelie (Fabuleux destin d'AmÃ©lie Poulain, Le)...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>1213</td>\n",
              "      <td>Goodfellas (1990)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>1206</td>\n",
              "      <td>Clockwork Orange, A (1971)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>593</td>\n",
              "      <td>Silence of the Lambs, The (1991)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>34</td>\n",
              "      <td>Babe (1995)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>480</td>\n",
              "      <td>Jurassic Park (1993)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>2571</td>\n",
              "      <td>Matrix, The (1999)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>592</td>\n",
              "      <td>Batman (1989)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>364</td>\n",
              "      <td>Lion King, The (1994)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>2028</td>\n",
              "      <td>Saving Private Ryan (1998)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>1580</td>\n",
              "      <td>Men in Black (a.k.a. MIB) (1997)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>648</td>\n",
              "      <td>Mission: Impossible (1996)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>7153</td>\n",
              "      <td>Lord of the Rings: The Return of the King, The...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>3793</td>\n",
              "      <td>X-Men (2000)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>6539</td>\n",
              "      <td>Pirates of the Caribbean: The Curse of the Bla...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>5349</td>\n",
              "      <td>Spider-Man (2002)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>1307</td>\n",
              "      <td>When Harry Met Sally... (1989)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>7438</td>\n",
              "      <td>Kill Bill: Vol. 2 (2004)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>8961</td>\n",
              "      <td>Incredibles, The (2004)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>2011</td>\n",
              "      <td>Back to the Future Part II (1989)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>58559</td>\n",
              "      <td>Dark Knight, The (2008)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>104</td>\n",
              "      <td>Happy Gilmore (1996)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>8368</td>\n",
              "      <td>Harry Potter and the Prisoner of Azkaban (2004)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>79132</td>\n",
              "      <td>Inception (2010)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>59315</td>\n",
              "      <td>Iron Man (2008)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>72998</td>\n",
              "      <td>Avatar (2009)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>69122</td>\n",
              "      <td>Hangover, The (2009)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>42</th>\n",
              "      <td>7293</td>\n",
              "      <td>50 First Dates (2004)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>3785</td>\n",
              "      <td>Scary Movie (2000)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>48385</td>\n",
              "      <td>Borat: Cultural Learnings of America for Make ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    MovieID                                              Title\n",
              "0        50                         Usual Suspects, The (1995)\n",
              "1       318                   Shawshank Redemption, The (1994)\n",
              "2       912                                  Casablanca (1942)\n",
              "3      2628   Star Wars: Episode I - The Phantom Menace (1999)\n",
              "4      1517  Austin Powers: International Man of Mystery (1...\n",
              "5       208                                  Waterworld (1995)\n",
              "6      2959                                  Fight Club (1999)\n",
              "7       780               Independence Day (a.k.a. ID4) (1996)\n",
              "8       736                                     Twister (1996)\n",
              "9      1198  Raiders of the Lost Ark (Indiana Jones and the...\n",
              "10      750  Dr. Strangelove or: How I Learned to Stop Worr...\n",
              "11     4226                                     Memento (2000)\n",
              "12     1197                         Princess Bride, The (1987)\n",
              "13     1721                                     Titanic (1997)\n",
              "14     1136             Monty Python and the Holy Grail (1975)\n",
              "15     4973  Amelie (Fabuleux destin d'AmÃ©lie Poulain, Le)...\n",
              "16     1213                                  Goodfellas (1990)\n",
              "17     1206                         Clockwork Orange, A (1971)\n",
              "18      593                   Silence of the Lambs, The (1991)\n",
              "19       34                                        Babe (1995)\n",
              "20      480                               Jurassic Park (1993)\n",
              "21     2571                                 Matrix, The (1999)\n",
              "22      592                                      Batman (1989)\n",
              "23      364                              Lion King, The (1994)\n",
              "24     2028                         Saving Private Ryan (1998)\n",
              "25     1580                   Men in Black (a.k.a. MIB) (1997)\n",
              "26      648                         Mission: Impossible (1996)\n",
              "27     7153  Lord of the Rings: The Return of the King, The...\n",
              "28     3793                                       X-Men (2000)\n",
              "29     6539  Pirates of the Caribbean: The Curse of the Bla...\n",
              "30     5349                                  Spider-Man (2002)\n",
              "31     1307                     When Harry Met Sally... (1989)\n",
              "32     7438                           Kill Bill: Vol. 2 (2004)\n",
              "33     8961                            Incredibles, The (2004)\n",
              "34     2011                  Back to the Future Part II (1989)\n",
              "35    58559                            Dark Knight, The (2008)\n",
              "36      104                               Happy Gilmore (1996)\n",
              "37     8368    Harry Potter and the Prisoner of Azkaban (2004)\n",
              "38    79132                                   Inception (2010)\n",
              "39    59315                                    Iron Man (2008)\n",
              "40    72998                                      Avatar (2009)\n",
              "41    69122                               Hangover, The (2009)\n",
              "42     7293                              50 First Dates (2004)\n",
              "43     3785                                 Scary Movie (2000)\n",
              "44    48385  Borat: Cultural Learnings of America for Make ..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 122
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vNbCzB9c6iUx",
        "colab_type": "text"
      },
      "source": [
        "### Adding Survey results to the data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yaeaYMpWjwJ6",
        "colab_type": "code",
        "outputId": "13e3ecfd-a891-42a7-d467-523836210cb0",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 75
        }
      },
      "source": [
        "#Loading Survey Data\n",
        "\n",
        "from google.colab import files\n",
        "uploaded = files.upload()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-37613474-bd52-43b9-9551-1e5b0206a4ff\" name=\"files[]\" multiple disabled />\n",
              "     <output id=\"result-37613474-bd52-43b9-9551-1e5b0206a4ff\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Saving movielens.xlsx to movielens.xlsx\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E2AsmtJiwJR-",
        "colab_type": "code",
        "outputId": "c0bed198-4151-4111-ced4-d3186a4bd8f5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 348
        }
      },
      "source": [
        "num_of_survey_ratings = picked_movies.shape[0]\n",
        "\n",
        "survey_data = pd.read_excel('movielens.xlsx')\n",
        "survey_data.drop(axis=0, index = 0, inplace=True)\n",
        "survey_data = survey_data.iloc[:, 12:].transpose()\n",
        "Names = survey_data.iloc[num_of_survey_ratings-1]\n",
        "survey_data.drop(survey_data.tail(4).index,inplace=True)\n",
        "survey_data.reset_index(inplace=True)\n",
        "survey_data.rename(columns={\"index\": \"Title\"}, inplace=True)\n",
        "survey_data['MovieID'] =  None\n",
        "survey_data.loc[:num_of_survey_ratings,'MovieID'] = picked_movies['MovieID']\n",
        "survey_data.MovieID.astype(int)\n",
        "\n",
        "##################################\n",
        "\n",
        "survey_data = survey_data.merge(picked_movies, how='left', on='MovieID')   \n",
        "# survey_data.Title = survey_data.Title.str.strip()\n",
        "survey_data.drop(['Title_y'], axis=1, inplace = True)\n",
        "survey_data.rename(columns={\"Title_x\": \"Title\"}, inplace=True)\n",
        "\n",
        "survey_data= pd.melt(survey_data, id_vars=['MovieID', 'Title'], value_vars=list(range(1,38)),\n",
        "                    var_name='UserID', value_name='rating')\n",
        "\n",
        "#dropping NAN ratings (where there was no rating in the survey)\n",
        "\n",
        "survey_data = survey_data[pd.notnull(survey_data['rating'])]\n",
        "survey_data = survey_data.reset_index(drop=True)\n",
        "\n",
        "survey_data.head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>MovieID</th>\n",
              "      <th>Title</th>\n",
              "      <th>UserID</th>\n",
              "      <th>rating</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>318</td>\n",
              "      <td>The Shawshank Redemption (1994)</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>912</td>\n",
              "      <td>Casablanca (1942)</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2628</td>\n",
              "      <td>Star Wars: Episode I - The Phantom Menace (1999)</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1517</td>\n",
              "      <td>Austin Powers: International Man of Mystery (1...</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>208</td>\n",
              "      <td>Waterworld (1995)</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>2959</td>\n",
              "      <td>Fight Club (1999)</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>780</td>\n",
              "      <td>Independence Day (a.k.a. ID4) (1996)</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>736</td>\n",
              "      <td>Twister (1996)</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1198</td>\n",
              "      <td>Raiders of the Lost Ark (Indiana Jones) (1981)</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>750</td>\n",
              "      <td>Dr. Strangelove (1964)</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  MovieID                                              Title UserID rating\n",
              "0     318                   The Shawshank Redemption (1994)       1      5\n",
              "1     912                                 Casablanca (1942)       1      3\n",
              "2    2628  Star Wars: Episode I - The Phantom Menace (1999)       1      2\n",
              "3    1517  Austin Powers: International Man of Mystery (1...      1      3\n",
              "4     208                                 Waterworld (1995)       1      2\n",
              "5    2959                                 Fight Club (1999)       1      5\n",
              "6     780              Independence Day (a.k.a. ID4) (1996)       1      3\n",
              "7     736                                    Twister (1996)       1      3\n",
              "8    1198    Raiders of the Lost Ark (Indiana Jones) (1981)       1      4\n",
              "9     750                             Dr. Strangelove (1964)      1      5"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 125
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "dAAPhn6n6tH0",
        "colab": {}
      },
      "source": [
        "user_ratings = survey_data[['UserID','MovieID','rating']]\n",
        "user_ratings.columns = ['u_id', 'i_id', 'rating']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jVC45he_hWsb",
        "colab_type": "text"
      },
      "source": [
        "Creating the dataset with addition of the survey users\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "SyHA2H7G6tH-",
        "colab": {}
      },
      "source": [
        "try:\n",
        "    movielens18 = movielens18.drop(columns=['timestamp'])\n",
        "except:\n",
        "    pass\n",
        "\n",
        "# shifting the u_id up, so the user_ids from the surveys will be 1,2,3 etc and the original u_id will start at 4, 5...\n",
        "n_survey_users = len(Names)\n",
        "# user_ratings.u_id += 1\n",
        "movielens18.u_id += n_survey_users\n",
        "\n",
        "data_with_user = user_ratings.append(movielens18, ignore_index=True).reset_index(drop=True)\n",
        "\n",
        "train_user = data_with_user.sample(frac=0.9)\n",
        "val_user = data_with_user.drop(train_user.index.tolist()).sample(frac=0.5, random_state=8)\n",
        "test_user = data_with_user.drop(train_user.index.tolist()).drop(val_user.index.tolist())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m5DOabm9b3Lt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data_with_user.groupby(by='u_id').count()[20:40]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7aYnXY1_cXZz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Crearing a popularity dataframe\n",
        "\n",
        "movie_popularity = data_with_user.groupby(by='i_id').count()\n",
        "movie_popularity['i_id'] = data_with_user.groupby(by='i_id').count().index\n",
        "movie_popularity.drop(['u_id'], axis=1, inplace=True)\n",
        "movie_popularity.rename(columns={\"rating\": \"popularity\"}, inplace=True)\n",
        "movie_popularity = movie_popularity.reset_index(drop=True)\n",
        "\n",
        "# merging movies_df with the popularity, by i_id\n",
        "\n",
        "movies_df_with_popularity = movies_df.merge(movie_popularity, how = 'left', left_on = 'i_id', right_on = 'i_id').reset_index(drop=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OtKijczuBwTp",
        "colab_type": "text"
      },
      "source": [
        "### Train + Getting predictions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "uFw8RF656tIH",
        "colab": {}
      },
      "source": [
        "from itertools import product\n",
        "from sklearn.metrics import mean_squared_error, mean_absolute_error\n",
        "def funk_svd_predict(userID, data_with_user, movies_df):\n",
        "    userID = [userID]\n",
        "\n",
        "    # all_users = data_with_user.u_id.unique()\n",
        "    all_movies = data_with_user.i_id.unique()\n",
        "    recommendations = pd.DataFrame(list(product(userID, all_movies)), columns=['u_id', 'i_id'])\n",
        "\n",
        "    #Getting predictions for the selected userID\n",
        "    pred_train = svd.predict(recommendations)\n",
        "    recommendations['prediction'] = pred_train\n",
        "    recommendations.head(10)\n",
        "\n",
        "    sorted_user_predictions = recommendations.sort_values(by='prediction', ascending=False)\n",
        "\n",
        "    user_ratings = data_with_user[data_with_user.u_id == userID[0]]\n",
        "    user_ratings.columns = ['u_id',\t'i_id', 'rating']\n",
        "    # Recommend the highest predicted rating movies that the user hasn't seen yet.\n",
        "    recommendations = movies_df[~movies_df['i_id'].isin(user_ratings['i_id'])].\\\n",
        "        merge(pd.DataFrame(sorted_user_predictions).reset_index(drop=True), how = 'inner', left_on = 'i_id', right_on = 'i_id').\\\n",
        "        sort_values(by='prediction', ascending = False).drop(['i_id'],axis=1)\n",
        "    recommendations = recommendations.reset_index(drop=True)\n",
        "\n",
        "    rated_df = movies_df[movies_df['i_id'].isin(user_ratings['i_id'])].\\\n",
        "        merge(pd.DataFrame(data_with_user).reset_index(drop=True), how = 'inner', left_on = 'i_id', right_on = 'i_id')\n",
        "    rated_df = rated_df.loc[rated_df.u_id==userID[0]].sort_values(by='rating', ascending = False)\n",
        "    rated_df = rated_df.reset_index(drop=True)\n",
        "    return recommendations, rated_df"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3bH9dXHZhdcq",
        "colab_type": "text"
      },
      "source": [
        "This are the best hyperparameters from our random search:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TcEfkZEp5fh2",
        "colab_type": "code",
        "outputId": "1864458a-7cf6-4b07-dc3c-b64e89905d01",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 526
        }
      },
      "source": [
        "lr, reg, factors = (0.007, 0.03, 90)\n",
        "epochs = 50\n",
        "\n",
        "svd = SVD(learning_rate=lr, regularization=reg, n_epochs=epochs, n_factors=factors,\n",
        "          min_rating=0.5, max_rating=5)\n",
        "\n",
        "svd.fit(X=train_user, X_val=val_user, early_stopping=True, shuffle=False)\n",
        "\n",
        "pred = svd.predict(test_user)\n",
        "mae = mean_absolute_error(test_user[\"rating\"], pred)\n",
        "rmse = np.sqrt(mean_squared_error(test_user[\"rating\"], pred))\n",
        "print(\"Test MAE:  {:.2f}\".format(mae))\n",
        "print(\"Test RMSE: {:.2f}\".format(rmse))\n",
        "print('{} factors, {} lr, {} reg'.format(factors, lr, reg))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Preprocessing data...\n",
            "\n",
            "Epoch 1/50  | val_loss: 0.76 - val_rmse: 0.87 - val_mae: 0.67 - took 8.0 sec\n",
            "Epoch 2/50  | val_loss: 0.74 - val_rmse: 0.86 - val_mae: 0.66 - took 7.6 sec\n",
            "Epoch 3/50  | val_loss: 0.72 - val_rmse: 0.85 - val_mae: 0.65 - took 7.4 sec\n",
            "Epoch 4/50  | val_loss: 0.70 - val_rmse: 0.84 - val_mae: 0.64 - took 7.2 sec\n",
            "Epoch 5/50  | val_loss: 0.69 - val_rmse: 0.83 - val_mae: 0.63 - took 7.4 sec\n",
            "Epoch 6/50  | val_loss: 0.67 - val_rmse: 0.82 - val_mae: 0.63 - took 7.3 sec\n",
            "Epoch 7/50  | val_loss: 0.66 - val_rmse: 0.81 - val_mae: 0.62 - took 7.4 sec\n",
            "Epoch 8/50  | val_loss: 0.65 - val_rmse: 0.81 - val_mae: 0.62 - took 7.5 sec\n",
            "Epoch 9/50  | val_loss: 0.64 - val_rmse: 0.80 - val_mae: 0.61 - took 7.6 sec\n",
            "Epoch 10/50 | val_loss: 0.64 - val_rmse: 0.80 - val_mae: 0.61 - took 7.6 sec\n",
            "Epoch 11/50 | val_loss: 0.63 - val_rmse: 0.79 - val_mae: 0.61 - took 7.5 sec\n",
            "Epoch 12/50 | val_loss: 0.62 - val_rmse: 0.79 - val_mae: 0.60 - took 7.3 sec\n",
            "Epoch 13/50 | val_loss: 0.62 - val_rmse: 0.79 - val_mae: 0.60 - took 7.5 sec\n",
            "Epoch 14/50 | val_loss: 0.62 - val_rmse: 0.78 - val_mae: 0.60 - took 7.2 sec\n",
            "Epoch 15/50 | val_loss: 0.61 - val_rmse: 0.78 - val_mae: 0.60 - took 7.3 sec\n",
            "Epoch 16/50 | val_loss: 0.61 - val_rmse: 0.78 - val_mae: 0.60 - took 7.3 sec\n",
            "Epoch 17/50 | val_loss: 0.61 - val_rmse: 0.78 - val_mae: 0.59 - took 7.4 sec\n",
            "Epoch 18/50 | val_loss: 0.60 - val_rmse: 0.78 - val_mae: 0.59 - took 7.4 sec\n",
            "Epoch 19/50 | val_loss: 0.60 - val_rmse: 0.78 - val_mae: 0.59 - took 7.3 sec\n",
            "Epoch 20/50 | val_loss: 0.60 - val_rmse: 0.77 - val_mae: 0.59 - took 7.4 sec\n",
            "Epoch 21/50 | val_loss: 0.60 - val_rmse: 0.77 - val_mae: 0.59 - took 7.3 sec\n",
            "\n",
            "Training took 3 min and 18 sec\n",
            "Test MAE:  0.59\n",
            "Test RMSE: 0.77\n",
            "90 factors, 0.007 lr, 0.03 reg\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eWMM_D93V88Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Survey users:\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ylzfm1AzhjgL",
        "colab_type": "text"
      },
      "source": [
        "### Getting recommendations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vTIWodsNhnvf",
        "colab_type": "text"
      },
      "source": [
        "A nice trick we added is slicing the recommendations for popular movies only.\n",
        "In the beggining, we got weird Soviet documentaries.. nobody wants to watch that.\n",
        "\n",
        "We deduced that this is because some random user in the 20M dataset has very similar rating pattern to the survey user, therefore we got results that coresponds to its taste.\n",
        "\n",
        "Decided to **recommend only popular movies (>100 ratings).** Reduced the variety and hurts discovery a bit,  but **improves results substantially**!\n",
        "\n",
        "**This is also done by the big guys (Netflix, Amazon, Youtube)**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xHpbDR3hAjPU",
        "colab_type": "code",
        "outputId": "2760c972-e58e-404e-f436-f9dda5641ba6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "Selected_user = 36\n",
        "print(\"predictions for {}\".format(Names[Selected_user]))\n",
        "\n",
        "recommendations, rated_df = funk_svd_predict(Selected_user, data_with_user, movies_df_with_popularity)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "predictions for Dean Shabi\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZJg7w9--BNTO",
        "colab_type": "code",
        "outputId": "5c5a958c-5d34-4154-911c-754b1bfb7d9a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 918
        }
      },
      "source": [
        "rated_df[['title', 'rating', 'genres']]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>title</th>\n",
              "      <th>rating</th>\n",
              "      <th>genres</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Inception (2010)</td>\n",
              "      <td>5</td>\n",
              "      <td>Action|Crime|Drama|Mystery|Sci-Fi|Thriller|IMAX</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Iron Man (2008)</td>\n",
              "      <td>5</td>\n",
              "      <td>Action|Adventure|Sci-Fi</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Lord of the Rings: The Return of the King, The...</td>\n",
              "      <td>5</td>\n",
              "      <td>Action|Adventure|Drama|Fantasy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Pirates of the Caribbean: The Curse of the Bla...</td>\n",
              "      <td>5</td>\n",
              "      <td>Action|Adventure|Comedy|Fantasy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Spider-Man (2002)</td>\n",
              "      <td>5</td>\n",
              "      <td>Action|Adventure|Sci-Fi|Thriller</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>X-Men (2000)</td>\n",
              "      <td>5</td>\n",
              "      <td>Action|Adventure|Sci-Fi</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>Scary Movie (2000)</td>\n",
              "      <td>5</td>\n",
              "      <td>Comedy|Horror</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Lion King, The (1994)</td>\n",
              "      <td>5</td>\n",
              "      <td>Adventure|Animation|Children|Drama|Musical|IMAX</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>Matrix, The (1999)</td>\n",
              "      <td>5</td>\n",
              "      <td>Action|Sci-Fi|Thriller</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>Incredibles, The (2004)</td>\n",
              "      <td>5</td>\n",
              "      <td>Action|Adventure|Animation|Children|Comedy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>Dark Knight, The (2008)</td>\n",
              "      <td>5</td>\n",
              "      <td>Action|Crime|Drama|IMAX</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>Titanic (1997)</td>\n",
              "      <td>5</td>\n",
              "      <td>Drama|Romance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>Batman (1989)</td>\n",
              "      <td>5</td>\n",
              "      <td>Action|Crime|Thriller</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>Harry Potter and the Prisoner of Azkaban (2004)</td>\n",
              "      <td>5</td>\n",
              "      <td>Adventure|Fantasy|IMAX</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>Men in Black (a.k.a. MIB) (1997)</td>\n",
              "      <td>4</td>\n",
              "      <td>Action|Comedy|Sci-Fi</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>Star Wars: Episode I - The Phantom Menace (1999)</td>\n",
              "      <td>4</td>\n",
              "      <td>Action|Adventure|Sci-Fi</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>Austin Powers: International Man of Mystery (1...</td>\n",
              "      <td>4</td>\n",
              "      <td>Action|Adventure|Comedy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>Hangover, The (2009)</td>\n",
              "      <td>4</td>\n",
              "      <td>Comedy|Crime</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>Mission: Impossible (1996)</td>\n",
              "      <td>4</td>\n",
              "      <td>Action|Adventure|Mystery|Thriller</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>Jurassic Park (1993)</td>\n",
              "      <td>4</td>\n",
              "      <td>Action|Adventure|Sci-Fi|Thriller</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>50 First Dates (2004)</td>\n",
              "      <td>4</td>\n",
              "      <td>Comedy|Romance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>Kill Bill: Vol. 2 (2004)</td>\n",
              "      <td>4</td>\n",
              "      <td>Action|Drama|Thriller</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>Avatar (2009)</td>\n",
              "      <td>3</td>\n",
              "      <td>Action|Adventure|Sci-Fi|IMAX</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>Happy Gilmore (1996)</td>\n",
              "      <td>3</td>\n",
              "      <td>Comedy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>Saving Private Ryan (1998)</td>\n",
              "      <td>3</td>\n",
              "      <td>Action|Drama|War</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>Back to the Future Part II (1989)</td>\n",
              "      <td>3</td>\n",
              "      <td>Adventure|Comedy|Sci-Fi</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>Raiders of the Lost Ark (Indiana Jones and the...</td>\n",
              "      <td>3</td>\n",
              "      <td>Action|Adventure</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>Monty Python and the Holy Grail (1975)</td>\n",
              "      <td>3</td>\n",
              "      <td>Adventure|Comedy|Fantasy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>Fight Club (1999)</td>\n",
              "      <td>3</td>\n",
              "      <td>Action|Crime|Drama|Thriller</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                title  ...                                           genres\n",
              "0                                    Inception (2010)  ...  Action|Crime|Drama|Mystery|Sci-Fi|Thriller|IMAX\n",
              "1                                     Iron Man (2008)  ...                          Action|Adventure|Sci-Fi\n",
              "2   Lord of the Rings: The Return of the King, The...  ...                   Action|Adventure|Drama|Fantasy\n",
              "3   Pirates of the Caribbean: The Curse of the Bla...  ...                  Action|Adventure|Comedy|Fantasy\n",
              "4                                   Spider-Man (2002)  ...                 Action|Adventure|Sci-Fi|Thriller\n",
              "5                                        X-Men (2000)  ...                          Action|Adventure|Sci-Fi\n",
              "6                                  Scary Movie (2000)  ...                                    Comedy|Horror\n",
              "7                               Lion King, The (1994)  ...  Adventure|Animation|Children|Drama|Musical|IMAX\n",
              "8                                  Matrix, The (1999)  ...                           Action|Sci-Fi|Thriller\n",
              "9                             Incredibles, The (2004)  ...       Action|Adventure|Animation|Children|Comedy\n",
              "10                            Dark Knight, The (2008)  ...                          Action|Crime|Drama|IMAX\n",
              "11                                     Titanic (1997)  ...                                    Drama|Romance\n",
              "12                                      Batman (1989)  ...                            Action|Crime|Thriller\n",
              "13    Harry Potter and the Prisoner of Azkaban (2004)  ...                           Adventure|Fantasy|IMAX\n",
              "14                   Men in Black (a.k.a. MIB) (1997)  ...                             Action|Comedy|Sci-Fi\n",
              "15   Star Wars: Episode I - The Phantom Menace (1999)  ...                          Action|Adventure|Sci-Fi\n",
              "16  Austin Powers: International Man of Mystery (1...  ...                          Action|Adventure|Comedy\n",
              "17                               Hangover, The (2009)  ...                                     Comedy|Crime\n",
              "18                         Mission: Impossible (1996)  ...                Action|Adventure|Mystery|Thriller\n",
              "19                               Jurassic Park (1993)  ...                 Action|Adventure|Sci-Fi|Thriller\n",
              "20                              50 First Dates (2004)  ...                                   Comedy|Romance\n",
              "21                           Kill Bill: Vol. 2 (2004)  ...                            Action|Drama|Thriller\n",
              "22                                      Avatar (2009)  ...                     Action|Adventure|Sci-Fi|IMAX\n",
              "23                               Happy Gilmore (1996)  ...                                           Comedy\n",
              "24                         Saving Private Ryan (1998)  ...                                 Action|Drama|War\n",
              "25                  Back to the Future Part II (1989)  ...                          Adventure|Comedy|Sci-Fi\n",
              "26  Raiders of the Lost Ark (Indiana Jones and the...  ...                                 Action|Adventure\n",
              "27             Monty Python and the Holy Grail (1975)  ...                         Adventure|Comedy|Fantasy\n",
              "28                                  Fight Club (1999)  ...                      Action|Crime|Drama|Thriller\n",
              "\n",
              "[29 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 153
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7fc9JngYBO4t",
        "colab_type": "code",
        "outputId": "c9d3ef3b-bbac-4cf5-945e-33ea53e6e9d6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "recommendations[['title', 'prediction', 'genres', 'popularity']][recommendations.popularity>=100].head(40)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>title</th>\n",
              "      <th>prediction</th>\n",
              "      <th>genres</th>\n",
              "      <th>popularity</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>North &amp; South (2004)</td>\n",
              "      <td>4.843947</td>\n",
              "      <td>Drama|Romance</td>\n",
              "      <td>147.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>3 Idiots (2009)</td>\n",
              "      <td>4.769312</td>\n",
              "      <td>Comedy|Drama|Romance</td>\n",
              "      <td>453.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>Shawshank Redemption, The (1994)</td>\n",
              "      <td>4.767195</td>\n",
              "      <td>Crime|Drama</td>\n",
              "      <td>63395.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Sixth Sense, The (1999)</td>\n",
              "      <td>4.757356</td>\n",
              "      <td>Drama|Horror|Mystery</td>\n",
              "      <td>39028.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>Lifted (2006)</td>\n",
              "      <td>4.748968</td>\n",
              "      <td>Animation|Comedy|Sci-Fi</td>\n",
              "      <td>127.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>Finding Nemo (2003)</td>\n",
              "      <td>4.727982</td>\n",
              "      <td>Adventure|Animation|Children|Comedy</td>\n",
              "      <td>23569.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>Shrek (2001)</td>\n",
              "      <td>4.717577</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy|Ro...</td>\n",
              "      <td>31972.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>Schindler's List (1993)</td>\n",
              "      <td>4.709197</td>\n",
              "      <td>Drama|War</td>\n",
              "      <td>50054.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>Harry Potter and the Deathly Hallows: Part 2 (...</td>\n",
              "      <td>4.704267</td>\n",
              "      <td>Action|Adventure|Drama|Fantasy|Mystery|IMAX</td>\n",
              "      <td>3983.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>Eddie Izzard: Dress to Kill (1999)</td>\n",
              "      <td>4.693521</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>208.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>The Fault in Our Stars (2014)</td>\n",
              "      <td>4.684287</td>\n",
              "      <td>Drama|Romance</td>\n",
              "      <td>376.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>Big Hero 6 (2014)</td>\n",
              "      <td>4.663239</td>\n",
              "      <td>Action|Animation|Comedy</td>\n",
              "      <td>448.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>Monsters, Inc. (2001)</td>\n",
              "      <td>4.649761</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
              "      <td>23657.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>Wallace &amp; Gromit: The Wrong Trousers (1993)</td>\n",
              "      <td>4.633727</td>\n",
              "      <td>Animation|Children|Comedy|Crime</td>\n",
              "      <td>15022.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>Auntie Mame (1958)</td>\n",
              "      <td>4.623021</td>\n",
              "      <td>Comedy|Drama</td>\n",
              "      <td>1333.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>Intouchables (2011)</td>\n",
              "      <td>4.609626</td>\n",
              "      <td>Comedy|Drama</td>\n",
              "      <td>2738.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>Battlestar Galactica (2003)</td>\n",
              "      <td>4.608633</td>\n",
              "      <td>Drama|Sci-Fi|War</td>\n",
              "      <td>2884.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>Good Will Hunting (1997)</td>\n",
              "      <td>4.608004</td>\n",
              "      <td>Drama|Romance</td>\n",
              "      <td>28324.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>Lord of the Rings: The Fellowship of the Ring,...</td>\n",
              "      <td>4.606756</td>\n",
              "      <td>Adventure|Fantasy</td>\n",
              "      <td>37553.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>Star Trek (2009)</td>\n",
              "      <td>4.600214</td>\n",
              "      <td>Action|Adventure|Sci-Fi|IMAX</td>\n",
              "      <td>7690.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>Toy Story 2 (1999)</td>\n",
              "      <td>4.598439</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
              "      <td>22770.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>Help, The (2011)</td>\n",
              "      <td>4.598273</td>\n",
              "      <td>Drama</td>\n",
              "      <td>1674.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>Anne of Green Gables (1985)</td>\n",
              "      <td>4.597808</td>\n",
              "      <td>Children|Drama</td>\n",
              "      <td>477.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>Toy Story 3 (2010)</td>\n",
              "      <td>4.595566</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy|IMAX</td>\n",
              "      <td>5781.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>Princess Bride, The (1987)</td>\n",
              "      <td>4.594741</td>\n",
              "      <td>Action|Adventure|Comedy|Fantasy|Romance</td>\n",
              "      <td>32608.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>42</th>\n",
              "      <td>Tangled (2010)</td>\n",
              "      <td>4.586490</td>\n",
              "      <td>Animation|Children|Comedy|Fantasy|Musical|Roma...</td>\n",
              "      <td>2438.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>Green Mile, The (1999)</td>\n",
              "      <td>4.585841</td>\n",
              "      <td>Crime|Drama</td>\n",
              "      <td>21080.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>Stardust (2007)</td>\n",
              "      <td>4.582819</td>\n",
              "      <td>Adventure|Comedy|Fantasy|Romance</td>\n",
              "      <td>4300.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>Usual Suspects, The (1995)</td>\n",
              "      <td>4.581229</td>\n",
              "      <td>Crime|Mystery|Thriller</td>\n",
              "      <td>47026.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>October Sky (1999)</td>\n",
              "      <td>4.580252</td>\n",
              "      <td>Drama</td>\n",
              "      <td>7378.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>Capitalism: A Love Story (2009)</td>\n",
              "      <td>4.580098</td>\n",
              "      <td>Documentary</td>\n",
              "      <td>458.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>51</th>\n",
              "      <td>Anne of Green Gables: The Sequel (a.k.a. Anne ...</td>\n",
              "      <td>4.576110</td>\n",
              "      <td>Children|Drama|Romance</td>\n",
              "      <td>179.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>Toy Story (1995)</td>\n",
              "      <td>4.575057</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
              "      <td>49695.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53</th>\n",
              "      <td>How to Train Your Dragon (2010)</td>\n",
              "      <td>4.572493</td>\n",
              "      <td>Adventure|Animation|Children|Fantasy|IMAX</td>\n",
              "      <td>4761.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>55</th>\n",
              "      <td>Kingsman: The Secret Service (2015)</td>\n",
              "      <td>4.570615</td>\n",
              "      <td>Action|Adventure|Comedy|Crime</td>\n",
              "      <td>159.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>Sound of Music, The (1965)</td>\n",
              "      <td>4.569106</td>\n",
              "      <td>Musical|Romance</td>\n",
              "      <td>14049.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57</th>\n",
              "      <td>Temple Grandin (2010)</td>\n",
              "      <td>4.567627</td>\n",
              "      <td>Drama</td>\n",
              "      <td>575.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>58</th>\n",
              "      <td>Harry Potter and the Deathly Hallows: Part 1 (...</td>\n",
              "      <td>4.567032</td>\n",
              "      <td>Action|Adventure|Fantasy|IMAX</td>\n",
              "      <td>3884.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>62</th>\n",
              "      <td>Hunger Games: Catching Fire, The (2013)</td>\n",
              "      <td>4.560270</td>\n",
              "      <td>Action|Adventure|Sci-Fi|IMAX</td>\n",
              "      <td>1873.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>64</th>\n",
              "      <td>Slumdog Millionaire (2008)</td>\n",
              "      <td>4.555823</td>\n",
              "      <td>Crime|Drama|Romance</td>\n",
              "      <td>9208.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                title  ...  popularity\n",
              "2                                North & South (2004)  ...       147.0\n",
              "5                                     3 Idiots (2009)  ...       453.0\n",
              "6                    Shawshank Redemption, The (1994)  ...     63395.0\n",
              "7                             Sixth Sense, The (1999)  ...     39028.0\n",
              "8                                       Lifted (2006)  ...       127.0\n",
              "10                                Finding Nemo (2003)  ...     23569.0\n",
              "11                                       Shrek (2001)  ...     31972.0\n",
              "13                            Schindler's List (1993)  ...     50054.0\n",
              "14  Harry Potter and the Deathly Hallows: Part 2 (...  ...      3983.0\n",
              "16                 Eddie Izzard: Dress to Kill (1999)  ...       208.0\n",
              "17                      The Fault in Our Stars (2014)  ...       376.0\n",
              "21                                  Big Hero 6 (2014)  ...       448.0\n",
              "22                              Monsters, Inc. (2001)  ...     23657.0\n",
              "24        Wallace & Gromit: The Wrong Trousers (1993)  ...     15022.0\n",
              "25                                 Auntie Mame (1958)  ...      1333.0\n",
              "29                                Intouchables (2011)  ...      2738.0\n",
              "30                        Battlestar Galactica (2003)  ...      2884.0\n",
              "31                           Good Will Hunting (1997)  ...     28324.0\n",
              "32  Lord of the Rings: The Fellowship of the Ring,...  ...     37553.0\n",
              "35                                   Star Trek (2009)  ...      7690.0\n",
              "36                                 Toy Story 2 (1999)  ...     22770.0\n",
              "37                                   Help, The (2011)  ...      1674.0\n",
              "38                        Anne of Green Gables (1985)  ...       477.0\n",
              "39                                 Toy Story 3 (2010)  ...      5781.0\n",
              "40                         Princess Bride, The (1987)  ...     32608.0\n",
              "42                                     Tangled (2010)  ...      2438.0\n",
              "43                             Green Mile, The (1999)  ...     21080.0\n",
              "44                                    Stardust (2007)  ...      4300.0\n",
              "45                         Usual Suspects, The (1995)  ...     47026.0\n",
              "49                                 October Sky (1999)  ...      7378.0\n",
              "50                    Capitalism: A Love Story (2009)  ...       458.0\n",
              "51  Anne of Green Gables: The Sequel (a.k.a. Anne ...  ...       179.0\n",
              "52                                   Toy Story (1995)  ...     49695.0\n",
              "53                    How to Train Your Dragon (2010)  ...      4761.0\n",
              "55                Kingsman: The Secret Service (2015)  ...       159.0\n",
              "56                         Sound of Music, The (1965)  ...     14049.0\n",
              "57                              Temple Grandin (2010)  ...       575.0\n",
              "58  Harry Potter and the Deathly Hallows: Part 1 (...  ...      3884.0\n",
              "62            Hunger Games: Catching Fire, The (2013)  ...      1873.0\n",
              "64                         Slumdog Millionaire (2008)  ...      9208.0\n",
              "\n",
              "[40 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 154
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "71GS99Z9fP7q",
        "colab_type": "text"
      },
      "source": [
        "### Export to Excell"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EYqlzGHsME2H",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install xlsxwriter\n",
        "import xlsxwriter\n",
        "\n",
        "writer = pd.ExcelWriter('predictions.xlsx', engine='xlsxwriter')\n",
        "\n",
        "for i, user in enumerate(Names):\n",
        "    print(\"recommendations for {}\".format(Names[i+1]))\n",
        "    recommendations, _ = funk_svd_predict(i+1, data_with_user, movies_df_with_popularity)\n",
        "    sheetName = str(Names[i+1])+str(i)\n",
        "    recommendations[['title', 'prediction', 'genres', 'popularity']][recommendations.popularity>=100].head(40).to_excel(writer, sheet_name=sheetName)\n",
        "writer.save()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8jVgT1xUeAyu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# !pip install xlsxwriter\n",
        "import xlsxwriter\n",
        "\n",
        "writer = pd.ExcelWriter('user_ratings.xlsx', engine='xlsxwriter')\n",
        "\n",
        "for i, user in enumerate(Names):\n",
        "    print(\"ratings for {}\".format(Names[i+1]))\n",
        "    _ , rated_df = funk_svd_predict(i+1, data_with_user, movies_df_with_popularity)\n",
        "    sheetName = str(Names[i+1])+str(i)\n",
        "    rated_df.to_excel(writer, sheet_name=sheetName)\n",
        "writer.save()\n"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}
